<!DOCTYPE html>
<html>

<head>

<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
<title>panda_class</title>


<style type="text/css">
body {
  font-family: Helvetica, arial, sans-serif;
  font-size: 14px;
  line-height: 1.6;
  padding-top: 10px;
  padding-bottom: 10px;
  background-color: white;
  padding: 30px; }

body > *:first-child {
  margin-top: 0 !important; }
body > *:last-child {
  margin-bottom: 0 !important; }

a {
  color: #4183C4; }
a.absent {
  color: #cc0000; }
a.anchor {
  display: block;
  padding-left: 30px;
  margin-left: -30px;
  cursor: pointer;
  position: absolute;
  top: 0;
  left: 0;
  bottom: 0; }

h1, h2, h3, h4, h5, h6 {
  margin: 20px 0 10px;
  padding: 0;
  font-weight: bold;
  -webkit-font-smoothing: antialiased;
  cursor: text;
  position: relative; }

h1:hover a.anchor, h2:hover a.anchor, h3:hover a.anchor, h4:hover a.anchor, h5:hover a.anchor, h6:hover a.anchor {
  background: url() no-repeat 10px center;
  text-decoration: none; }

h1 tt, h1 code {
  font-size: inherit; }

h2 tt, h2 code {
  font-size: inherit; }

h3 tt, h3 code {
  font-size: inherit; }

h4 tt, h4 code {
  font-size: inherit; }

h5 tt, h5 code {
  font-size: inherit; }

h6 tt, h6 code {
  font-size: inherit; }

h1 {
  font-size: 28px;
  color: black; }

h2 {
  font-size: 24px;
  border-bottom: 1px solid #cccccc;
  color: black; }

h3 {
  font-size: 18px; }

h4 {
  font-size: 16px; }

h5 {
  font-size: 14px; }

h6 {
  color: #777777;
  font-size: 14px; }

p, blockquote, ul, ol, dl, li, table, pre {
  margin: 15px 0; }

hr {
  background: transparent url() repeat-x 0 0;
  border: 0 none;
  color: #cccccc;
  height: 4px;
  padding: 0;
}

body > h2:first-child {
  margin-top: 0;
  padding-top: 0; }
body > h1:first-child {
  margin-top: 0;
  padding-top: 0; }
  body > h1:first-child + h2 {
    margin-top: 0;
    padding-top: 0; }
body > h3:first-child, body > h4:first-child, body > h5:first-child, body > h6:first-child {
  margin-top: 0;
  padding-top: 0; }

a:first-child h1, a:first-child h2, a:first-child h3, a:first-child h4, a:first-child h5, a:first-child h6 {
  margin-top: 0;
  padding-top: 0; }

h1 p, h2 p, h3 p, h4 p, h5 p, h6 p {
  margin-top: 0; }

li p.first {
  display: inline-block; }
li {
  margin: 0; }
ul, ol {
  padding-left: 30px; }

ul :first-child, ol :first-child {
  margin-top: 0; }

dl {
  padding: 0; }
  dl dt {
    font-size: 14px;
    font-weight: bold;
    font-style: italic;
    padding: 0;
    margin: 15px 0 5px; }
    dl dt:first-child {
      padding: 0; }
    dl dt > :first-child {
      margin-top: 0; }
    dl dt > :last-child {
      margin-bottom: 0; }
  dl dd {
    margin: 0 0 15px;
    padding: 0 15px; }
    dl dd > :first-child {
      margin-top: 0; }
    dl dd > :last-child {
      margin-bottom: 0; }

blockquote {
  border-left: 4px solid #dddddd;
  padding: 0 15px;
  color: #777777; }
  blockquote > :first-child {
    margin-top: 0; }
  blockquote > :last-child {
    margin-bottom: 0; }

table {
  padding: 0;border-collapse: collapse; }
  table tr {
    border-top: 1px solid #cccccc;
    background-color: white;
    margin: 0;
    padding: 0; }
    table tr:nth-child(2n) {
      background-color: #f8f8f8; }
    table tr th {
      font-weight: bold;
      border: 1px solid #cccccc;
      margin: 0;
      padding: 6px 13px; }
    table tr td {
      border: 1px solid #cccccc;
      margin: 0;
      padding: 6px 13px; }
    table tr th :first-child, table tr td :first-child {
      margin-top: 0; }
    table tr th :last-child, table tr td :last-child {
      margin-bottom: 0; }

img {
  max-width: 100%; }

span.frame {
  display: block;
  overflow: hidden; }
  span.frame > span {
    border: 1px solid #dddddd;
    display: block;
    float: left;
    overflow: hidden;
    margin: 13px 0 0;
    padding: 7px;
    width: auto; }
  span.frame span img {
    display: block;
    float: left; }
  span.frame span span {
    clear: both;
    color: #333333;
    display: block;
    padding: 5px 0 0; }
span.align-center {
  display: block;
  overflow: hidden;
  clear: both; }
  span.align-center > span {
    display: block;
    overflow: hidden;
    margin: 13px auto 0;
    text-align: center; }
  span.align-center span img {
    margin: 0 auto;
    text-align: center; }
span.align-right {
  display: block;
  overflow: hidden;
  clear: both; }
  span.align-right > span {
    display: block;
    overflow: hidden;
    margin: 13px 0 0;
    text-align: right; }
  span.align-right span img {
    margin: 0;
    text-align: right; }
span.float-left {
  display: block;
  margin-right: 13px;
  overflow: hidden;
  float: left; }
  span.float-left span {
    margin: 13px 0 0; }
span.float-right {
  display: block;
  margin-left: 13px;
  overflow: hidden;
  float: right; }
  span.float-right > span {
    display: block;
    overflow: hidden;
    margin: 13px auto 0;
    text-align: right; }

code, tt {
  margin: 0 2px;
  padding: 0 5px;
  white-space: nowrap;
  border: 1px solid #eaeaea;
  background-color: #f8f8f8;
  border-radius: 3px; }

pre code {
  margin: 0;
  padding: 0;
  white-space: pre;
  border: none;
  background: transparent; }

.highlight pre {
  background-color: #f8f8f8;
  border: 1px solid #cccccc;
  font-size: 13px;
  line-height: 19px;
  overflow: auto;
  padding: 6px 10px;
  border-radius: 3px; }

pre {
  background-color: #f8f8f8;
  border: 1px solid #cccccc;
  font-size: 13px;
  line-height: 19px;
  overflow: auto;
  padding: 6px 10px;
  border-radius: 3px; }
  pre code, pre tt {
    background-color: transparent;
    border: none; }

sup {
    font-size: 0.83em;
    vertical-align: super;
    line-height: 0;
}

kbd {
  display: inline-block;
  padding: 3px 5px;
  font-size: 11px;
  line-height: 10px;
  color: #555;
  vertical-align: middle;
  background-color: #fcfcfc;
  border: solid 1px #ccc;
  border-bottom-color: #bbb;
  border-radius: 3px;
  box-shadow: inset 0 -1px 0 #bbb
}

* {
	-webkit-print-color-adjust: exact;
}
@media screen and (min-width: 914px) {
    body {
        width: 854px;
        margin:0 auto;
    }
}
@media print {
	table, pre {
		page-break-inside: avoid;
	}
	pre {
		word-wrap: break-word;
	}
}
</style>


</head>

<body>

<h1 id="toc_0">PANDA Grand Tour</h1>

<p>Tim Leek <code>tleek@ll.mit.edu</code></p>

<h2 id="toc_1">Introduction</h2>

<p>In this class, you will use PANDA to investigate the dynamic behavior of program code and the operating system that contains it. PANDA is a whole-system dynamic analysis platform based upon the Qemu emulator. An entire operating system runs under PANDA, which makes it especially useful for situations in which concurrent processes and even kernel activity must be understood.</p>

<p>Further, execution can be recorded and replayed, which is a powerful ability. Various analysis plugins that run on the execution replay allow visibility into the emulated guest and provide information about what is going on there. Reverse Engineering with PANDA is typically an iterative process, replaying a recorded execution over and over under different plugins and inspecting output to accumulate better and more complete understanding of behavior.</p>

<p><img src="images/pandalogo.png" style="width: 50%"/></p>

<h2 id="toc_2">Overview</h2>

<p>This course is introductory in nature. We will use PANDA and the set of existing plugins to examine, in detail, goings on inside an operating system. You will gain experience in recording and replaying system activity involving multiple processes and the kernel. You will make use of operating system introspection to track processes, decode system calls, and monitor file system activity. You will identify instructions of interest using memory taps. You will pull network activity and executables from recordings for analysis. You will use PANDA&#39;S dynamic taint system to follow crucial data flows.</p>

<p>Note that this course does not involve any coding. That is, developing PANDA plugins is beyond the scope of this course. This course is about getting a feel for what kinds of things PANDA can do already. PANDA&#39;s plugins are written in C or C++. There is information online (especially in the PANDA manual) that will help you get started developing plugins. </p>

<hr>

<h2 id="toc_3">Lab Prerequisites</h2>

<h3 id="toc_4">A shell</h3>

<p>You will need a shell in which to type commands. On OSX, this is the included <code>Terminal</code> program or something like <code>iTerm</code>. On Windows, this is the <code>CMD</code> window or perhaps powershell. On Linux, this is a terminal, e.g., <code>xterm</code> or <code>gnome-terminal</code>. </p>

<p>NOTE: for the remainder of this class, we will refer to all of these various command-entering thingeys as the <em>shell</em>.</p>

<h3 id="toc_5">Class Materials</h3>

<p>You&#39;ll need a bunch of support files to take this class.  Everything in the <code>panda_class_materials</code> directory. The class organizer will tell you how to get this either from a USB drive being passed around the class or from a network file share or the internet or whatever. Please copy that directory into your <em>home</em> directory. If you are on Linux or OSX, this will mean the materials ends up in <code>~/panda_class_materials</code>.  If you are on Windows, they should be in <code>C:\User\You</code>.</p>

<h3 id="toc_6">Docker</h3>

<p>You will be running PANDA inside a <code>docker</code> container. If you don&#39;t already have docker installed, you will need to get it.  </p>

<ol>
<li>For OSX or Windows, install docker desktop: <a href="https://docs.docker.com/docker-for-mac/install/">osx docker desktop</a> or <a href="%5Bhttps://docs.docker.com/docker-for-windows/">win docker desktop</a>.</li>
<li>For Linux, install with something like <code>apt-get install docker</code>. Basically, if you are running Linux you probably don&#39;t need to be told how to install docker; you came out of the womb knowing how to do this.</li>
</ol>

<p>For windows, you may need to log out and log back in after installing docker desktop.      </p>

<hr>

<h3 id="toc_7">PANDA Docker image</h3>

<p>All of our PANDA work will be done from a shell inside a docker container.  There is a pre-built, exported PANDA docker image in that <code>panda_class_materials</code> directory you copied earlier. You can resurrect it now, which takes less time than building from the <code>Dockerfile</code>.  </p>

<p><strong>CUE #1: Load the PANDA docker image</strong> </p>

<p>Here&#39;s how to do that on OSX or Linux:</p>

<div><pre><code class="language-none">docker load &lt; ~/panda_class_materials/docker/panda_docker.tgz</code></pre></div>

<p>And here&#39;s how to do it in Windows on Powershell:</p>

<div><pre><code class="language-none">PS C:\Users\You&gt; docker load -i C:\Users\You\panda_class_materials\docker\panda_docker.tgz</code></pre></div>

<p>If this succeeds you should see output that looks like this.</p>

<div><pre><code class="language-none">6cebf3abed5f: Loading layer [==================================================&gt;]  65.56MB/65.56MB
f7eae43028b3: Loading layer [==================================================&gt;]  991.2kB/991.2kB
7beb13bce073: Loading layer [==================================================&gt;]  15.87kB/15.87kB
122be11ab4a2: Loading layer [==================================================&gt;]  3.072kB/3.072kB
00cd1153e007: Loading layer [==================================================&gt;]  5.632kB/5.632kB
f72103c0a681: Loading layer [==================================================&gt;]  5.632kB/5.632kB
aa7f78dcc893: Loading layer [==================================================&gt;]  5.632kB/5.632kB
41725f77d683: Loading layer [==================================================&gt;]  75.71MB/75.71MB
6706076c6db1: Loading layer [==================================================&gt;]  1.326GB/1.326GB
375d7b0bd913: Loading layer [==================================================&gt;]   5.12kB/5.12kB
affb9719df40: Loading layer [==================================================&gt;]  297.5MB/297.5MB
708977fb29c8: Loading layer [==================================================&gt;]  1.272GB/1.272GB
Loaded image: panda:latest</code></pre></div>

<hr>

<h2 id="toc_8">PANDA First Gear -- A Whole Operating System At Our Command</h2>

<p>In this section of the class, we will boot an entire OS under PANDA and play around with it a bit. First we need a shell inside the PANDA Docker container.</p>

<p>On OSX or Linux, here&#39;s how to get that PANDA docker shell.</p>

<div><pre><code class="language-none">% docker run -it -v ~/panda_class_materials:/panda_class_materials panda bash</code></pre></div>

<p>On Windows, here&#39;s how to get that PANDA docker shell:</p>

<div><pre><code class="language-none">PS C:\Users\You&gt; docker run -it -v C:\User\You\panda_class_materials:/panda_class_materials panda bash</code></pre></div>

<p>The <code>-v</code> part, here, makes the class materials directory available inside the docker container at <code>/panda_class_materials</code>.  Here&#39;s what you should see if you get a PANDA docker shell.</p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# </code></pre></div>

<p>Now, from within that docker container, we will use PANDA to boot an operating system.</p>

<p><strong>CUE #2: Boot Linux with PANDA</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# /panda/build/i386-softmmu/panda-system-i386 -nographic /panda_class_materials/qcows/wheezy.qcow2 </code></pre></div>

<p>After a few seconds, you should see the Linux <code>grub</code> boot screen. Just wait a little longer and the Linux operating system will start to boot. </p>

<p>You will see lots of log messages scroll by as the Linux guest boots:</p>

<p><strong>CUE #3: What booting Linux under PANDA looks like</strong></p>

<div><pre><code class="language-none">Loading Linux 3.2.0-4-686-pae ...
Loading initial ramdisk ...
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.2.0-4-686-pae (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.51-1
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
[    0.000000]  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 0000000007fe0000 (usable)
[    0.000000]  BIOS-e820: 0000000007fe0000 - 0000000008000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
[    0.000000] Notice: NX (Execute Disable) protection missing in CPU!
[    0.000000] SMBIOS 2.8 present.
...</code></pre></div>

<p>When boot is complete, you should see a login prompt.  </p>

<div><pre><code class="language-none">Debian GNU/Linux 7 debian-i386 ttyS0

debian-i386 login: </code></pre></div>

<p><strong>CUE #4: How long did Linux take to boot with PANDA?</strong></p>

<p>For apples-to-pears refererence, booting a 64-bit more modern version of Linux under Vmware Fusion takes about 20 seconds. Emulation slows things down. </p>

<p><strong>CUE #5: Log into that Linux guest.</strong></p>

<p>Finally, go ahead and log in with username <code>root</code> and password <code>root</code>. You should see a prompt line like this:</p>

<div><pre><code class="language-none">    root@debian-i386:~# </code></pre></div>

<h2 id="toc_9">PANDA Second Gear -- Record and Replay All The Things</h2>

<p>In this section of the class, we are going to create a recording of something happening in the guest operating system. This is a core PANDA capability. Recordings are usefule because</p>

<ol>
<li>Recordings can be replayed over and over and the same sequence of instructions execute in the same order every time, with pointer addresses, process ids, and so on all being the same. This enables iterative dynamic reversing in which you can build up knowledge.</li>
<li>A recording can be replayed with as much and as heavy instrumentation or analysis as desired.  This is not true of live execution. </li>
<li>Recordings can capture weird, ephemeral, or dangerous activity, such as the activity of malware or anti-virus. </li>
<li>Recordings can be shared with other reverse engineers and results will correspond: the same processes with the same PIDs and all the same pointer addresses.</li>
</ol>

<p>Whilst interacting <em>live</em> with a PANDA instance, we can create a recording of whole-system activity using its <em>monitor</em>. The monitor allows you to interact with the emulation engine via a command-line interface (if you are familiar with Qemu, this monitor is simply Qemu&#39;s monitor augmented with additional PANDA commands). There is a lot you can do from the monitor: halt, resume, snapshot or restore a guest, inspect registers or memory, plug in a USB device of CD, etc.   </p>

<p>Return to that docker shell in which you ran PANDA and booted Linux. Make sure you are logged in.  </p>

<p>First things first, you are going to need to know how to enter and exit the monitor. It&#39;s very simple. </p>

<p><strong>CUE #6: To enter or exit the monitor, type <code>Ctrl-a</code>, followed by the letter <code>c</code>.</strong></p>

<p>When you enter the monitor, you will see the following prompt: <code>(qemu)</code>. That&#39;s how you can tell that you are in the monitor. Try this a few times right now to get the hang of it: switching back and forth between the guest Linux OS and the monitor. Note: you&#39;ll have to hit enter in the Linux guest to see its prompt.</p>

<p>Now let&#39;s create a PANDA recording.  Switch to the monitor and tell PANDA to begin a recording named &quot;commands&quot; by typing the following.</p>

<p><strong>CUE #7: How to start a recording using the monitor</strong></p>

<div><pre><code class="language-none">(qemu) begin_record commands</code></pre></div>

<p>PANDA should respond with </p>

<div><pre><code class="language-none">(qemu) writing snapshot:    ./commands-rr-snp
opening nondet log for write :  ./commands-rr-nondet.log</code></pre></div>

<p>Now switch from the monitor back to the Linux guest.</p>

<div><pre><code class="language-none">root@debian-i386:~# </code></pre></div>

<p>Next, execute the following three commands, in sequence, in the guest.</p>

<p><strong>CUE #8: The commands you are to enter into the PANDA Linux guest</strong></p>

<ol>
<li><code>ls -lt /etc</code></li>
<li><code>ps -ef</code></li>
<li><code>netstat -a</code></li>
</ol>

<p>Lastly, return to the monitor and end the recording</p>

<p><strong>CUE #9: How to end a recording using the monitor</strong></p>

<div><pre><code class="language-none">(qemu) end_record</code></pre></div>

<p>To which, PANDA will respond with something like</p>

<div><pre><code class="language-none">(qemu) Time taken was: 29 seconds.
Checksum of guest memory: 0xd4acf418</code></pre></div>

<p>At this point, you can shut down that Linux guest you booted under PANDA with the following command. </p>

<div><pre><code class="language-none">root@debian-i386:~# /sbin/shutdown -h now</code></pre></div>

<p>Shutdown should take about 10-15 seconds, after which, you will get your docker shell prompt back. Congratulations! You just created a recording of a whole lot of whole-system activity with PANDA. </p>

<p>You can replay all that activity with the following command from the docker shell. </p>

<p><strong>CUE #10: How to get PANDA to replay the execution of those commands you entered into the guest</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# /panda/build/i386-softmmu/panda-system-i386 -replay commands </code></pre></div>

<p>You will see output indicating replay progress: percent of nondeterminism log replayed, instruction count, etc. It will look something like this.</p>

<p><strong>CUE #11: What PANDA replay looks like</strong></p>

<div><pre><code class="language-none">loading snapshot
... done.
opening nondet log for read :   ./commands-rr-nondet.log
./commands-rr-nondet.log:    34403335 instrs total.
commands:      344035 (  1.00%) instrs.    0.18 sec.  0.15 GB ram.
commands:      688073 (  2.00%) instrs.    0.20 sec.  0.15 GB ram.
commands:     1032103 (  3.00%) instrs.    0.22 sec.  0.15 GB ram.
commands:     1376139 (  4.00%) instrs.    0.26 sec.  0.15 GB ram.
commands:     1720168 (  5.00%) instrs.    0.30 sec.  0.16 GB ram.
commands:     2064203 (  6.00%) instrs.    0.34 sec.  0.16 GB ram.
...
commands:    32339138 ( 94.00%) instrs.    1.82 sec.  0.18 GB ram.
commands:    32683170 ( 95.00%) instrs.    1.83 sec.  0.18 GB ram.
commands:    33027203 ( 96.00%) instrs.    1.84 sec.  0.18 GB ram.
commands:    33371239 ( 97.00%) instrs.    1.86 sec.  0.18 GB ram.
commands:    33715271 ( 98.00%) instrs.    1.87 sec.  0.18 GB ram.
commands:    34059308 ( 99.00%) instrs.    1.88 sec.  0.18 GB ram.
./commands-rr-nondet.log:  log is empty.
./commands-rr-nondet.log:  log is empty.
Replay completed successfully. 1
Time taken was: 2 seconds.
Stats:
RR_INPUT_1 number = 0, size = 0 bytes
RR_INPUT_2 number = 0, size = 0 bytes
RR_INPUT_4 number = 6909, size = 96726 bytes
RR_INPUT_8 number = 13942, size = 250956 bytes
RR_INTERRUPT_REQUEST number = 3753, size = 52542 bytes
RR_EXIT_REQUEST number = 0, size = 0 bytes
RR_SKIPPED_CALL number = 29079, size = 908845 bytes
RR_END_OF_LOG number = 1, size = 10 bytes
RR_PENDING_INTERRUPTS number = 0, size = 0 bytes
RR_EXCEPTION number = 0, size = 0 bytes
max_queue_len = 193
Checksum of guest memory: 0x173e5d95
Replay completed successfully 2.</code></pre></div>

<p><strong>CUE #12: How long does it take to replay <code>commands</code>? Anything weird about that?</strong></p>

<p>The specifics of your replay will likely be different. In the one pictured above, 1.88 seconds and over 34 million instructions were required to replay everything that happened, which is the execution of those three commands in sequence. That may seem a lot. That&#39;s because it is. Remember that replay included all program, library, and kernel code that executed to get the business of those commands done. It also included all the background activity, including daemons and the kernel doing its thing.</p>

<p>Take a look at the size of the replay you just created. It consists of two files, really:</p>

<ol>
<li><code>commands-rr-snp</code>: a snapshot of the state of the operating system at the time the recording begain (RAM + registers + a little more)</li>
<li><code>commands-rr-nondet.log</code>: the log of nondeterministic inputs that has to be replayed</li>
</ol>

<p>In my case, the snapshot is about 80MB and the nondeterminism log is about 1.3MB. This means we need just over 80MB to completely specify a perfect recording of over 28M instructions. Practically, what this means is that sharing replays is easy and common.</p>

<p>There are simple things we can do to peek into that replay, to get more out of it than just how many instructions executed. For example, we can add a little to the command-line we just used and PANDA will spit out all the code that was executed plus a little more information.</p>

<p><strong>CUE #13: How to get PANDA to spit out all the code that executes during a replay</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# /panda/build/i386-softmmu/panda-system-i386 -replay commands -d in_asm,rr,int |&amp; less
loading snapshot
... done.
opening nondet log for read :   ./commands-rr-nondet.log
Servicing hardware INT=0xef
     0: v=ef e=0000 i=0 cpl=0 IP=0060:c10243a0 pc=c10243a0 SP=0068:c13dbfd0 env-&gt;regs[R_EAX]=00000000
EAX=00000000 EBX=c14170ac ECX=c7ed0f80 EDX=00000003
ESI=00000000 EDI=c13de000 EBP=01989003 ESP=c13dbfd0
EIP=c10243a0 EFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =007b 00000000 ffffffff 00cff300 DPL=3 DS   [-WA]
CS =0060 00000000 ffffffff 00cf9a00 DPL=0 CS32 [-R-]
SS =0068 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =007b 00000000 ffffffff 00cff300 DPL=3 DS   [-WA]
FS =00d8 06a54000 ffffffff 008f9300 DPL=0 DS16 [-WA]
GS =00e0 c7ed5940 00000018 00409100 DPL=0 DS   [--A]
LDT=0000 00000000 00000000 00008200 DPL=0 LDT
TR =0080 c7ed3780 0000206b 00008900 DPL=0 TSS32-avl
GDT=     c7ece000 000000ff
IDT=     c13de000 000007ff
CR0=8005003b CR2=080ebed0 CR3=06d28000 CR4=000006b0
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
CCS=00000044 CCD=00000000 CCO=EFLAGS  
EFER=0000000000000000
----------------
IN: 
0xc12c4648:  push   0xffffff10
0xc12c464d:  cld    
0xc12c464e:  push   gs
0xc12c4650:  push   fs
0xc12c4652:  push   es
0xc12c4653:  push   ds
0xc12c4654:  push   eax
0xc12c4655:  push   ebp
0xc12c4656:  push   edi
0xc12c4657:  push   esi
0xc12c4658:  push   edx
0xc12c4659:  push   ecx
0xc12c465a:  push   ebx
0xc12c465b:  mov    edx,0x7b
0xc12c4660:  mov    ds,edx

Prog point: 0xc12c4648 {guest_instr_count=0}
Prog point: 0xc12c4648 {guest_instr_count=0}
----------------
IN: 
0xc12c4662:  mov    es,edx

Prog point: 0xc12c4662 {guest_instr_count=15}
...</code></pre></div>

<p>You can page through this for a while and you will see blocks of x86 code disassembled. If you were very patient you could page through all of it. But there is a lot -- over 7 million lines for the replay pictured above.</p>

<p><strong>CUE #14: Discussion about why this isn&#39;t more like 28M lines</strong></p>

<p>That kind of output seems like it would be useful, and sometimes it is. Mostly it isn&#39;t. That&#39;s because there is simply too much of it and it is very low level without any context. What instructions correspond to the <code>ls</code>, <code>ps</code>, and <code>netstat</code> processes? We have no idea. Currently, our view of this replay is just the sequence of machine instructions plus a few other things like when interrupts and exceptions happen.</p>

<h2 id="toc_10">PANDA Third Gear -- Operating System Introspection</h2>

<p>Let&#39;s use some PANDA plugins to get a little more out of that replay. First, we&#39;ll use a stock plugin called <code>asidstory</code>, so named because it generates an ASCII art picture of the temporal <em>story</em> of what happened during a replay, in terms of the <em>asid</em>s of the processes that executed.  <code>asid</code> stands for _Address _Space _IDentifier. This is the contents of a specific machine register that points to the virtual page tables for a process. The <code>asid</code> identifies a process uniquely. Here is the command line you can use to generate that ASCII art asidstory.</p>

<p><strong>CUE #15: Run your commands reply with the <code>asidstory</code> plugin using this command-line</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# /panda/build/i386-softmmu/panda-system-i386 -replay commands -os linux-32-debian:3.2.0-4-686-pae -panda asidstory</code></pre></div>

<p>You should see the replay scroll by as before, albeit slightly slower, after which, you can inspect the output by looking at the file <code>asidstory</code>. Your output should look somewhat like the following.</p>

<div><pre><code class="language-none">root@ac6aadd740c8: cat asidstory
   Count   Pid                  Name      Asid     First          Last
     220  2522                    ps   7997000   1751123  -&gt;  13770450
     199  2523                    ls   79a6000  16578881  -&gt;  26047264
     125  2524               netstat   6f3b000  27759956  -&gt;  33622531
      99  2480                  bash   78cf000    137879  -&gt;  34004657
      10  2522                 bash2   6f06000   1385108  -&gt;   1620983
...
swapper/02 : [#                                                                          ]
      cron : [#                                                                          ]
      bash : [#####           ####                                #####                  ]
      init : [ # #                                                   #                   ]
     bash2 : [    #                                                                      ]
     bash3 : [    ##                                                                     ]
        ls : [     ############                                                          ]
       ls2 : [                #                                                          ]
     bash4 : [                   ##                                                      ]
     bash5 : [                    #                                                      ]
        ps : [                    #################################                      ]
       ps2 : [                                                    #                      ]
     bash6 : [                                                        ##                 ]
     bash7 : [                                                         #                 ]
   netstat : [                                                         ################# ]
</code></pre></div>

<p>The vertical axis has a row for each process identified during the replay. The horizontal axis of this &quot;plot&quot; is time. The far left corresponds to the start of the replay and the far right is the end of the replay. Time is divided up into a number of cells. A hash mark, #, in a cell indicates that a process was seen to be executing at least once during that interval of the replay. If you scan back to the sequence of commands you executed, you should see them reflected in that same sequence in your <code>asidstory</code>.  First, <code>ls</code> executes, then we see <code>ps</code>, and, finally, <code>netstat</code> runs. The shell, <code>bash</code>, executes in between.  </p>

<p>This richer understanding of what is going on is made possible by <em>operating system introspection</em> (OSI). The plugin <code>asidstory</code> requires it (invisibly) and this is why part of the command line provides PANDA with information about the guest OS: the kernel version, the CPU and whether this OS is 32 or 64 bits. OSI is used all over the place in PANDA plugins as it is so useful. In this case, it allows the code for <code>asidstory</code> to be written once, in a few hundred lines of C++, in an operating system neutral way. The same plugin code works for Windows OS replays.</p>

<p>The top part of the output of <code>asidstory</code> provides more precise information. You can determine the range of instructions over which any process was observed during the replay, e.g. You can also see the PID and <code>asid</code> as well as a count of the number of times each process was observed. Note that the three commands you ran by hand make up the lion&#39;s share of the replay.</p>

<h2 id="toc_11">PANDA Fourth Gear -- Dynamic String Search</h2>

<p>In this section, you will use the <code>stringsearch</code> PANDA plugin to do a little light reverse engineering. </p>

<p><code>stringsearch</code> allows you to identify exactly what code performs some calculation, given that you can control or at least predict either the input or output of that calculation. This might mean you can control the password typed in to a program (even if you don&#39;t know the password). Or it might mean you know that the uncompressed mp3 data for Rick Astley&#39;s &quot;Never Gonna Give You Up&quot; will be output and thus played out of your speaker. </p>

<p>You first create a PANDA recording in which that input or output is guaranteed, by construction, to be read from or written to memory at some point. Then, you replay the recording with the <code>stringsearch</code> plugin, providing, as a parameter, the known data. The plugin reports any string matches it encounters, corresponding to it having seen that data read or written. </p>

<p>What good is all this? Sometimes, merely isolating the code performing some calculation is a big reverse engineering win. For instance, precisely identifying the code that de-obfuscates a string in malware, or that encrypts exfiltrated data. Another use for <code>stringsearch</code> is building introspection gadgets without requiring any domain knowledge or any reverse engineering work. That&#39;s what we are going to do now. </p>

<p>You will recall that, when we booted that Linux OS image before, it output a bunch of messages. These are useful, since they provide insight into what subsystem is being set up when. However, if you took a recording of boot and tried to play it back, you wouldn&#39;t have them. Or, if someone handed you a recording purportedly of boot and you replayed it, you&#39;d just see replay progress messages. And forget about using <code>asidstory</code> since it won&#39;t tell you much of use until boot is nearly complete. There are no processes during boot, just a kernel.  </p>

<p>There is just such a recording in <code>panda_class_materials/replays/boot</code>.  </p>

<p><strong>CUE #16: Use what you&#39;ve learned so far to replay that <code>boot</code> recording.  Be annoyed by the fact that you don&#39;t have boot messages.</strong></p>

<p>Now scroll back in your terminal to see some of those boot messages. Note that some of them are pretty unique. Choose one, preferably near the beginning of the replay, and replay with something like the following commands.</p>

<p><strong>CUE #17: Replaying using the <code>stringsearch</code> plugin</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# /panda/build/i386-softmmu/panda-system-i386 -replay /panda_class_materials/replays/boot -panda stringsearch:str=&quot;Linux version 3.2.0-4-686-pae&quot;</code></pre></div>

<p>You should see output something like this.</p>

<div><pre><code class="language-none">...
boot:    47432553 (  1.00%) instrs.    5.27 sec.  0.09 GB ram.
boot:    94865100 (  2.00%) instrs.   10.55 sec.  0.09 GB ram.
boot:   142297662 (  3.00%) instrs.   16.11 sec.  0.10 GB ram.
boot:   189730201 (  4.00%) instrs.   21.48 sec.  0.10 GB ram.
boot:   237162751 (  5.00%) instrs.   27.07 sec.  0.11 GB ram.
boot:   284595306 (  6.00%) instrs.   31.08 sec.  0.11 GB ram.
boot:   332027849 (  7.00%) instrs.   35.12 sec.  0.11 GB ram.
READ Match of str 0 at: instr_count=346437193 :  01510e00 0150edb8 (asid=0x00000000)
WRITE Match of str 0 at: instr_count=346437193 :  01510e00 0150edb8 (asid=0x00000000)
READ Match of str 0 at: instr_count=347091916 :  c116354a c1165818 (asid=0x00000000)
READ Match of str 0 at: instr_count=347092742 :  c1164178 c1163572 (asid=0x00000000)
WRITE Match of str 0 at: instr_count=347092743 :  c1164178 c1163575 (asid=0x00000000)
READ Match of str 0 at: instr_count=347095970 :  c12bf1e5 c10390c0 (asid=0x00000000)
READ Match of str 0 at: instr_count=347095974 :  c12bf1e5 c10390a8 (asid=0x00000000)
WRITE Match of str 0 at: instr_count=347095980 :  c10390b0 c1038840 (asid=0x00000000)
READ Match of str 0 at: instr_count=347095999 :  c12bf1e5 c10390b0 (asid=0x00000000)
READ Match of str 0 at: instr_count=347100561 :  c103912c c1038bdf (asid=0x00000000)
READ Match of str 0 at: instr_count=369467817 :  c1039776 c1038bdf (asid=0x00000000)
READ Match of str 0 at: instr_count=369471989 :  c1038654 c11dd4d9 (asid=0x00000000)
WRITE Match of str 0 at: instr_count=369471993 :  c1038654 c11dd4e2 (asid=0x00000000)
READ Match of str 0 at: instr_count=369471998 :  c1038654 c11dd4f1 (asid=0x00000000)
READ Match of str 0 at: instr_count=369472004 :  c1038654 c11dd5c5 (asid=0x00000000)
READ Match of str 0 at: instr_count=369472008 :  c1038654 c11dd5d5 (asid=0x00000000)
READ Match of str 0 at: instr_count=369999831 :  c1039776 c1038bdf (asid=0x00000000)
READ Match of str 0 at: instr_count=370003178 :  c11e43e9 c11e076a (asid=0x00000000)
READ Match of str 0 at: instr_count=370003180 :  c11e43e9 c11e077b (asid=0x00000000)
boot:   379460398 (  8.00%) instrs.   42.67 sec.  0.16 GB ram.
boot:   426892948 (  9.00%) instrs.   53.04 sec.  0.18 GB ram.
...</code></pre></div>

<p>Yes, you can stop the replay after you have seen that big block of read/write matches and start seeing just the percent progress indicators. You won&#39;t see any more matches.</p>

<p>What that output is telling you is that <code>stringsearch</code> worked as advertised. It found a bunch of replay instruction counts at which the string you gave it to search for was observed passing through a memory read or write instruction. The hex numbers near the end of each match are the program counter of the read or write instruction and the first address on the callstack, indicating some of the context from which this read or write instruction was executed.  Note that what we really have here in the output of <code>stringsearch</code> is a trace of the data-flow journey this particular log message experienced. Probably the later matches correspond to kernel print function handling of the string which we may want to hook as our gadget. </p>

<p>If we were being scientific, we might run <code>stringsearch</code> twice, with two different seemingly unique log messages and look for the intersection, since that would be more likely to contain the boot log tap point we want. In this way, we might tease apart the different dataflow journeys of the different messages and find where they converge, thus indicating a good tap point for extracting boot logs.</p>

<p>Now we are going to use the <code>stringsearch</code> output to fashion a boot log extractor that will work on this or any other replay from this operating system image. We will use the <code>textprinter</code> plugin (a companion to <code>stringsearch</code>) which logs anything read or written by the instructions you specify in a magic file: <code>tap_points.txt</code>. Create that file, now, and put the following in it, which identifies that last read match as the one from which to extract log messages.</p>

<p><strong>CUE #18: Create <code>tap_points.txt</code> file required by <code>textprinter</code></strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# cat &gt; tap_points.txt
0
c11e43e9 c11e077b
Ctrl-C</code></pre></div>

<p>And then run the <code>textprinter</code> plugin on your replay.</p>

<p><strong>CUE #19: Replay using <code>textprinter</code> to collect tap point data, i.e., boot log messages.</strong> </p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# /panda/build/i386-softmmu/panda-system-i386 -replay /panda_class_materials/replays/boot -panda textprinter</code></pre></div>

<p>You can safely halt replay at round 25% by pressing <code>Ctrl-C</code>.  Don&#39;t worry you will still get plenty of log messages. The output of <code>textprinter</code> is in the file <code>read_tap_buffers.txt.gz</code> which you can look at with <code>zcat</code>, if you like, but it is a bit hard to follow. Instead, we&#39;ll use the <code>render_log.py</code> script provided in the <code>panda_class_materials</code> to reconstruct the log.  Here&#39;s the contents of that script, in case you are curious.  It&#39;s quite simple.  </p>

<div><pre><code class="language-none">import sys
import fileinput
for line in fileinput.input():
    parts = line.split()
    c = int(parts[-1], 16)
    if c&gt;0 and c&lt;256:
        sys.stdout.write(chr(c))</code></pre></div>

<p><strong>CUE #20: Reconstruct boot logs using script <code>render_log.py</code>.</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# zcat read_tap_buffers.txt.gz | python /panda_class_materials/scripts/render_log.py | less</code></pre></div>

<p>Congratulations again! You just built your first introspection gadget for PANDA. </p>

<h2 id="toc_12">PANDA Fifth Gear -- Reverse Engineering</h2>

<p>As we noted earlier, PANDA&#39;s record/replay allows one to share interesting whole-system activity with another reverse engineer. There is just such a recording in <code>/panda_class_materials/replays/commands_wtf</code>. This was created by a colleague trying out an early version of the activities in this class. She followed the directions above to create a recording of executing those three commands, in sequence, just as you did earlier. However, she was concerned about the output she got from the <code>asidstory</code> plugin. Let&#39;s look into it.  </p>

<p><strong>CUE #21: Run PANDA with <code>asidstory</code> on the <code>commands_wtf</code> replay now.</strong></p>

<p>Now take a look at the output in the file <code>asidstory</code>. It looks pretty different from before, right? You can still see the three commands running: <code>ls</code>, <code>ps</code>, and <code>netstat</code>. But there&#39;s a some periodic activity layered on top and a bunch more <code>bash</code> processes than before. We also see a program called <code>ignorame</code> running, which sounds weird. Plus a few we may find familiar: <code>nc</code> and <code>sleep</code>. Ignoring the bash output we can see a little clearer what is going on. </p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# grep -v bash asidstory </code></pre></div>

<p><strong>CUE #22: Does this output look at all worrying to you?</strong></p>

<p>Does anyone have a theory about what UNIX pattern might have caused this output?</p>

<p>Let&#39;s peer a little closer into the activity of the new program  <code>ignorame</code> using a plugin called <code>syshist</code>, which outputs a histogram of counts for system calls, by asid.</p>

<p><strong>CUE #23: Running PANDA with the <code>syshist</code> plugin.</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# /panda/build/i386-softmmu/panda-system-i386 -replay /panda_class_materials/replays/commands_wtf -os linux-32-debian:3.2.0-4-686-pae -panda syscalls2:load-info=1 -panda syshist</code></pre></div>

<p>This plugin, like <code>asidstory</code>, produces output in a file of its own name. Let&#39;s look at the subset of output for the asid of the 1st <code>ignorame</code> process. First we need to find that process&#39; asid.</p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# grep &#39;ignorame &#39; asidstory
    60  2788              ignorame   57d0000   3778010  -&gt;   7484008
          ignorame : [    ######                                                                 ]
                                                        ]</code></pre></div>

<p>That 4th column is the <code>asid</code> for the 1st <code>ignorame</code> process, which we can use pull out just the <code>syshist</code> output for <code>ignorame</code>.</p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# grep 57d0000 syshist 
57d0000 sys_read 2
57d0000 sys_write 2
57d0000 sys_open 4
57d0000 sys_close 2
57d0000 sys_access 3
57d0000 sys_brk 3
57d0000 sys_munmap 1
57d0000 sys_mprotect 4
57d0000 sys_getdents 2
57d0000 sys_mmap_pgoff 7
57d0000 sys_stat64 4
57d0000 sys_fstat64 3
57d0000 sys_set_thread_area 1
57d0000 sys_exit_group 1</code></pre></div>

<p>This is one window into what <code>ignorame</code> does. It reads and writes files, reads directory and file information, asks for memory. The number after each call is a count of the number of times that system call was observed. So its not making tons of calls.  <code>ignorame</code> seems pretty, actually. Well maybe.  </p>

<p>We can get a little more insight into what is going on by using the <code>filemon</code> plugin which monitors reads and writes to files. You can probably guess where its output goes: into a file called <code>filemon</code>.</p>

<p><strong>CUE #24: Running PANDA with the <code>filemon</code> plugin.</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# /panda/build/i386-softmmu/panda-system-i386 -replay /panda_class_materials/replays/commands_wtf -os linux-32-debian:3.2.0-4-686-pae -panda filemon</code></pre></div>

<p>This plugin will log all file read and writes. We can restrict our attention to just output for that first version of the process <code>ignorame</code> as follows.  </p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# grep 2788 filemon 
read-enter-2788-2788-3-ignorame-26 &quot;/lib/i386-linux-gnu/i686/cmov/libc-2.13.so&quot;
read-return-2788-2788-3-ignorame-26
read-enter-2788-2788-4-ignorame-31 &quot;/home/joeuser/.ssh/id_rsa&quot;
read-return-2788-2788-4-ignorame-31
write-enter-2788-2788-1-ignorame-34 &quot;pipe:&quot;
write-enter-2788-2788-1-ignorame-35 &quot;pipe:&quot;
write-enter-2788-2788-1-ignorame-36 &quot;pipe:&quot;
write-enter-2788-2788-1-ignorame-37 &quot;pipe:&quot;</code></pre></div>

<p>Ok we are getting somewhere. This output is pretty sketchy. The program <code>ignorame</code> appears to be reading someone&#39;s ssh private key and then writing something to a pipe. But wait, there&#39;s more.  The plugin <code>filemon</code> <em>captures</em> the contents of those read/writes, in files named <code>read-return-2788-2788-4-ignorame-31</code> etc.</p>

<p><strong>CUE #25: Examine the file reads and writes by <code>ignorame</code> pid 2788 and hypothesize what might be going on.</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# cat read-return-2788-2788-4-ignorame-31
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA0KXYGCTbeIgv+ljVh4QoiskVjJhG0+L+hudlTkOzpE3w8YSP
SKUdx5jWsxe4I/8JthcCqMA/zvbIGWIE5ciYmwGgqnGQiEyzRmHk39857FrxM8gL
oinRpIQz0oJxwNLqGDoPBKjk/mvXyL35shK/tK1SyhNCT04J37wKdd0sZCTCju9W
gjseM3inEmm6CXymC2twAEkqcnKc4+Zme8CBvc1wH3UJPaJt657X+Z/pDNX1aH7S
DgCVzuFtrGMT6LbzHs5WUBkuOiR+CumL8QH2RbsXlBVw/6RSqjCg7s+WLdx1vViR
fQnOq5Wd35ZsiPtTgXxOx0gnklD3ga5jpJbELwIDAQABAoIBAHEGxi8eGD4NGGB3
qwSEVSoJkEDfZM73koYl570j87a3+iP7eVsTBzTO2M+fs8LcL7iScoBT1L0a9n65
mImZxwGBOjXa2z7avesMsN2NoWUmRq+UWKj6GnUSoLgSreiU2PdSzDvr+lDQvs4i
...
root@ac6aadd740c8:/# cat write-enter-2788-2788-1-ignorame-35
00 31 33 19 39 01 14 
73 7b 41 67 49 24 27 2d 17 18 4c 18 37 27 42 3a 0c 1f 3a 0b 30 23 42 21 1b 0f 41 67 49 4b 4f 60 13 1f 25 0f 14 27 2b 28 1f 17 27 09 25 37 27 2b 6e 1d 34 13 23 25 36 08 3b 1f 0b 3c 4f 0a 08 3c 36 62 3d 25 0d 15 09 3c 34 1c 04 0d 54 4d 2e 41 36 23 08 26 30 0d 2d 10 2e 13 5f 3d 5c 3f 31 3a 54 05 27 1f 00 1e 57 00 09 25 14 2f 50 2f 4d 52 14 22 04 29 27 17 2f 2b 71 2c 1a 28 2d 21 35 23 1b 63 0f 23 3d 0b 15 2d 39 27 02 0d 35 0f 27 13 24 04 01 02 0f 55 5b 52 6b 61 2a 38 1c 2b 5a 0d 12 5c 03 23 0a 34 12 23 0f 2c 5c 25 2e 1e 15 24 12 27 2b 0e 0b 36 20 21 34 3d 43 27 12 3e 1b 26 6d 63 1f 22 2f 49 16 21 6f 05 15 22 2a 25 36 5a 6a 1c 5f 7d 13 2d 06 0e 6e 25 36 09 30 25 08 1f 67 01 66 2d 0e 15 07 27 6d 3f 02 0f 09 0b 54 29 
06 2f 01 09 56 12 15 2b 1b 3d 1d 29 0a 2d 01 5e 75 0c 01 2f 5c 25 20 1c 3d 67 1b 02 57 33 
...</code></pre></div>

<p>Well we certainly see the ssh private key being read. And we also see a bunch of hex numbers separated by spaces being written to a pipe. </p>

<p>Note that we also know that <code>nc</code> is running periodically, from our inspection of the <code>asidstory</code> output. This means something might be being written to the network. We can use PANDA&#39;s <code>network</code> plugin to inspect those writes. This plugin sends its output, in binary form, to a <code>pcap</code> file.</p>

<p><strong>CUE #26: Run PANDA with the <code>network</code> plugin.</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# /panda/build/i386-softmmu/panda-system-i386 -replay /panda_class_materials/replays/commands_wtf -panda network:file=commands_wtf.pcap.gz</code></pre></div>

<p>You&#39;ll have to decompress that output.</p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# gzip -d commands_wtf.pcap.gz</code></pre></div>

<p>One way of looking at this network capture is with <code>tcpick</code>:</p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# tcpick -C -yP -r commands_wtf.pcap | less -R
Timeout for connections is 600
tcpick: reading from commands_wtf.pcap
1      SYN-SENT       10.0.2.15:36637 &gt; 192.168.80.1:5144
1      SYN-RECEIVED   10.0.2.15:36637 &gt; 192.168.80.1:5144
1      ESTABLISHED    10.0.2.15:36637 &gt; 192.168.80.1:5144
00 31 33 19 39 01 14 
73 7b 41 67 49 24 27 2d 17 18 4c 18 37 27 42 3a 0c 1f 3a 0b 30 23 42 21 1b 0f 41 67 49 4b 4f 60 13 1f 25 0f 14 27 2b 28 1f 17 27 09 25 37 27 2b 6e 1d 34 13 23 25 36 08 3b 1f 0b 3c 4f 0a 08 3c 36 62 3d 25 0d 15 09 3c 34 1c 04 0d 54 4d 2e 41 36 23 08 26 30 0d 2d 10 2e 13 5f 3d 5c 3f 31 3a 54 05 27 1f 00 1e 57 00 09 25 14 2f 50 2f 4d 52 14 22 04 29 27 17 2f 2b 71 2c 1a 28 2d 21 35 23 1b 63 0f 23 3d 0b 15 2d 39 27 02 0d 35 0f 27 13 24 04 01 02 0f 55 5b 52 6b 61 2a 38 1c 2b 5a 0d 12 5c 03 23 0a 34 12 23 0f 2c 5c 25 2e 1e 15 24 12 27 2b 0e 0b 36 20 21 34 3d 43 27 12 3e 1b 26 6d 63 1f 22 2f 49 16 21 6f 05 15 22 2a 25 36 5a 6a 1c 5f 7d 13 2d 06 0e 6e 25 36 09 30 25 08 1f 67 01 66 2d 0e 15 07 27 6d 3f 02 0f 09 0b 54 29 06 2f 01 09 56 12 15 2b 1b 3d 1d 29 0a 2d 01 5e 75 0c 01 2f 5c 25 20 1c 3d 67 1b 02 57 33 28 3a 3f 1c 18 7c 51 51 3a 41 04 79 1c 0e 2a 3e 53 0b 16 61 3f 40 20 01 21 3c 24 23 2a 3e 16 21 2f 3e 68 1a 0e 30 2c 15 57 3d 0b 14 07 3f 2b 0f 30 41 1d 23 01 06 5c 37 2a 58 0c 34 1f 12 08 24 34 1d 71 60 3e 19 15 0c 21 0d 69 25 47 1d 28 02 1a 5b 28 00 05 18 6e 00 33 04 11 27 59 1d 00 55 57 30 2d 3f 3c 3e 30 01 3a 12 11 2e 5c 2d 0a 0d 0e 2e 6d 31 0d 7f 0e 16 28 08 1b 1a 1b 03 20 27 33 2b 1c 17 03 03 26 27 2a 2f 19 2e 05 72 01 21 26 5e 10 11 2b 08 57 6c 13 1d 0d 13 3a 19 0b 2c 09 2f 1a 30 36 07 53 55 09 05 07 3a 59 7d 54 0c 5a 5d 3f 65 47 23 34 51 07 3c 2d 02 2e 30 30 29 50 27 75 30 1f 72 28 05 2e 5d 37 05 0f 25 26 32 53 2
...</code></pre></div>

<p><strong>CUE #27: Examine this output.  Anything look familiar?</strong></p>

<p>That&#39;s right. The output of <code>ignorame</code> in the form of those hex numbers is getting sent out on the network. The <code>network</code> plugin actually provides a little more info in the form of comments attached to packets. These comments indicate the instruction count at which network packets went out. If we examine this pcap using <code>less</code>, we see a lot of garbage but also what is clearly those hex numbers being output on the network and, most important, we can see where in the replay this is happening.</p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# less commands_wtf.pcap 
...
...Guest instruction count: 9369201...
...
... 00 31 33 19 39 01 14 
73 7b 41 67 49 24 27 2d 17 18 4c 18 37 27 42 3a 0c 1f 3a 0b 30 23 42 21 1b 0f 41 67 49 4b 4f 60 13 1f 25 0f 14 27 2b 28 1f 17 27 09 25 37 27 2b 6e 1d 34 13 23 25 36 08 3b 1f 0b 3c 4f 0a 08 3c 36 62 3d 25 0d 15 09 3c 34 1c 04 0d 54 4d 2e 41 36 23 08 26 30 0d 2d 10 2e 13 5f 3d 5c 3f 31 3a 54 05 27 1f 00 1e 57 00 09 25 14 2f 50 2f 4d 52 14 22 04 29 27 17 2f 2b 71 2c 1a 28 2d 21 35 23 1b ...</code></pre></div>

<p>This tells us that the first time a bunch of <code>ignorame</code>&#39;s output goes out on the network is at instruction count 9369201.</p>

<p><strong>CUE #28: See if you can use the output of <code>asidstory</code> to figure out which process is running at that instruction count.</strong></p>

<p>If you aren&#39;t good at comparing large numbers, there&#39;s obviously a better way to do this, which is to write a python script to analyze that <code>asidstory</code> output, in other words to determine which processes were operating at a particular instruction count. Such a script is provided in the class materials directory, which you should study at your leisure. For now, here&#39;s how to use it.</p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# python /panda_class_materials/scripts/find_procs_at_instr_count.py 9369201 ./asidstory 
      84  2789                   nc2   784c000   4184337  -&gt;  10062168
      81  2772                  bash   6f20000    168458  -&gt;  59360811
      73  2444                 bash4   533d000   2100845  -&gt;  52321733
       9     1                  init   6d28000   2071794  -&gt;  52468776</code></pre></div>

<p>The <code>bash</code> and <code>init</code> processes aren&#39;t likely to be the culprit. So that means <code>nc</code> is probably the one writing out those hex numbers on the network.  </p>

<p><strong>CUE #29: <code>filemon</code> captured reads and writes for all processes.  You may also want to examine what it collected for <code>nc</code>.</strong></p>

<p>So to sum up, we know that</p>

<ol>
<li><code>ignorame</code> reads <code>id_rsa</code>, which is naughty</li>
<li><code>ignorame</code> writes a bunch of hex numbers separated by spaces to a pipe</li>
<li>Sadly, these hex numbers aren&#39;t just the ASCII for the private key.  They look like junk.</li>
<li><code>nc</code> writes those same hex numbers out on the network</li>
</ol>

<p>One explanation that fits these facts is that <code>ignorame</code> is encrypting <code>id_rsa</code> before sending it out on the network. We have two ways of exploring this possibility with PANDA. The first involves extracting the binary from the replay and then spending time doing static analysis. PANDA&#39;s <code>memsavep</code> plugin can dump the guest&#39;s physical memory at a particular replay instruction. You can use this dump with another tool (not part of PANDA) called <code>volatility</code> (built for forensic analysis) in order to extract the binary for <code>ignorame</code>. We aren&#39;t going to pursue that avenue in this class since it veers off into static analysis and our bailywick is dynamic analysis. If you are interested in that kind of thing, we can reccommend the <code>Ghidra</code>, which you can obtain here: <a href="https://www.nsa.gov/resources/everyone/ghidra/">https://www.nsa.gov/resources/everyone/ghidra/</a>.</p>

<h3 id="toc_13">PANDA Top Gear (Ludicrous Speed) -- Dynamic Taint Analysis</h3>

<p>This piece of malware, <code>ignorame</code> happens to be tiny and quite simple. But what if it weren&#39;t? What if it were a few megabytes worth of deliberately convoluted code of which only a small portion were of concern to you? Say you wanted to understand the encryption used on <code>id_rsa</code> before sending it out on the network?</p>

<p>PANDA has a sophisticated dynamic taint analysis built in that can help here. The taint system allows you to apply labels to data in a replay and then track those labels as data flows through the system. Labels are tracked through copies, between processes, into and out of the kernel. When computation happens involving tainted data, the system collects sets of labels indicating derivation from taint sources. </p>

<p>Let&#39;s make this concrete by using PANDA&#39;s taint to analyze this encryption a little. However, taint analysis is slow since it is powerful and precise. It will slow down your replay by 20-40x. Thus, first, we will want to pull out a small portion of the replay to work with. Let&#39;s use PANDA&#39;s <code>scissors</code> plugin to do that. We merely specify the start and end instruction count (from <code>asidstory</code>) and a name for the new replay files. Probably you want to subtract a few 10s of thousands of instructions from the start instruction and add a few tens of thousands to the end instruction, just to be safe.</p>

<p><strong>CUE #30: Use PANDA <code>scissors</code> plugin to extract a smaller section replay.</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# /panda/build/i386-softmmu/panda-system-i386 -replay /panda_class_materials/replays/commands_wtf -panda scissors:name=ignorame,start=3778010,end=7484008</code></pre></div>

<p>This should create replay files in the current directory called <code>ignorame</code>. We are now going to use two plugins that rely upon the <code>taint2</code> plugin&#39;s underlying taint analysis.  <code>file_taint</code> will provide us the ability to apply taint labels to a particular file being read. The first byte in the file gets the label <code>1</code>, the second gets the label <code>2</code>, which we call <em>positional</em> labels. The `tainted_instr plugin identifies and logs all instructions that use tainted data. </p>

<p><strong>CUE #31: Replay with <code>file_taint</code> and <code>tainted_instr</code> plugins.</strong> </p>

<div><pre><code class="language-none"> root@ac6aadd740c8:/# /panda/build/i386-softmmu/panda-system-i386 -replay ignorame -os linux-32-debian:3.2.0-4-686-pae -pandalog ignorame.plog  -panda file_taint:filename=id_rsa,pos=1 -panda tainted_instr</code></pre></div>

<p>The output of <code>tainted_instr</code> goes in a file named <code>ignorame.plog</code>. This log is in the <code>pandalog</code> format, which uses Google protocol buffers under the hood but adds compression as well as the ability to rewind and fast forward. You can examine this log directly with the script <code>plog_reader.py</code>:</p>

<p><strong>CUE #32: Simple pandalog viewing with <code>plog_reader.py</code></strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# /panda/panda/scripts/plog_reader.py ./ignorame.plog | less
{
  &quot;pc&quot;: &quot;3077637219&quot;, 
  &quot;taintedInstr&quot;: {
    &quot;callStack&quot;: {
      &quot;addr&quot;: [
        &quot;3077637219&quot;, 
        &quot;134514782&quot;, 
        &quot;3076918854&quot;, 
        &quot;134513937&quot;
      ]
    }, 
    &quot;taintQuery&quot;: [
      {
        &quot;tcn&quot;: 0, 
        &quot;uniqueLabelSet&quot;: {
          &quot;ptr&quot;: &quot;140220515913728&quot;, 
          &quot;label&quot;: [
            0
          ]
        }, 
        &quot;ptr&quot;: &quot;140220515913728&quot;, 
        &quot;offset&quot;: 0
      }
    ]
  }, 
  &quot;instr&quot;: &quot;928658&quot;
},
   </code></pre></div>

<p>It isn&#39;t advisable to use <code>plog_reader.py</code> to peruse this log, since it is a lot of information. Here, it generates tens of millions of lines of output. Instead, one typically writes a small script that uses the class <code>PLogReader</code> to navigate the log, collect, filter, and summarize its output. There is such a script in <code>/panda_class_materials/scripts/taint.py</code> which you should run now. </p>

<p><strong>CUE #33: Pandalog analysis via script.</strong></p>

<div><pre><code class="language-none">root@ac6aadd740c8:/# python /panda_class_materials/scripts/taint.py ./ignorame.plog 
tainted instr @ pc=80486f1 count=327405
tainted instr @ pc=804872e count=9444
tainted instr @ pc=8048738 count=66111
tainted instr @ pc=8048757 count=100740
tainted instr @ pc=8048774 count=75555
tainted instr @ pc=80487f8 count=21827
tainted instr @ pc=b768c030 count=10074
tainted instr @ pc=b768c0d0 count=74850
tainted instr @ pc=b768d824 count=13432
tainted instr @ pc=b768f960 count=8395
tainted instr @ pc=b769010b count=16790
tainted instr @ pc=b76b9995 count=8390
tainted instr @ pc=b76b99a0 count=6570
tainted instr @ pc=b76bc438 count=5
tainted instr @ pc=b76bc440 count=5
tainted instr @ pc=b7713463 count=1679
tainted instr @ pc=c10d2b78 count=11
tainted instr @ pc=c1165e4d count=10
tainted instr @ pc=c1165e56 count=7971</code></pre></div>

<p>The immediate output of this script is the set of instructions seen to process tainted data, i.e., the contents of <code>id_rsa</code>. Those whose program counter start with <code>0x8</code> are part of <code>ignorame</code>. Those that start with <code>0xb</code> are library code. And those that start with <code>0xc</code> are kernel code. Thus, only six instructions in <code>ignorame</code> actually involve tainted data. This should narrow reverse engineering down quite a bit.  </p>

<p>Another output of the script is a dump of the so-called <em>taint compute number</em> or <em>tcn</em> as a function of instruction count. This is a measure of how computationally distant (depth of the tree of computation) a tainted result is from its taint source. This is output in the file <code>tcns</code> in this current directory in two columns: instruction count and tcn. We can plot this in something like R to see how the tcn changes over time to gain insight.  </p>

<p>One final note.  The script to process this pandalog is a less than thirty lines thanks to the nice abstraction of protocol buffers.  Here it is in its entirety. </p>

<div><pre><code class="language-none">import sys
import itertools
from google.protobuf.json_format import MessageToJson

sys.path.append(&#39;/panda/panda/scripts&#39;)

from plog_reader import PLogReader

f = open(&quot;tcns&quot;, &quot;w&quot;)
pcc = {}
with PLogReader(sys.argv[1]) as plr:
    for i,m in enumerate(plr):
        if m.HasField(&#39;tainted_instr&#39;):
            if not m.pc in pcc:
                pcc[m.pc] = 0
            pcc[m.pc] += 1
            for tb in m.tainted_instr.taint_query:
                f.write(&quot;%d %d\n&quot; % (m.instr, tb.tcn))

f.close()

pcs = list(pcc.keys())
pcs.sort()

for pc in pcs:
    print &quot;tainted instr @ pc=%x count=%d&quot; % (pc, pcc[pc])</code></pre></div>

<h3 id="toc_14">PANDA: Plaid Mode</h3>

<p>The real power of PANDA is in writing plugins. This requires programming in C, C++, or (coming soon!) Python. We won&#39;t cover that in this class, but you are encourage to read the PANDA manual, and read the code used to implement the various plugins you used in this class to get started. Plugins live in <code>/panda/panda/plugins/NAME_OF_PLUGIN</code>.  </p>

<p>Feel free to get in touch if you have questions, accolades, or complaints about PANDA.  </p>




</body>

</html>
